Last week, NovaSky, a team of researchers at the Sky Computing Lab at the University of California, Berkeley, released Sky-T1-32B-Preview, a reasoning model that effectively competes with previous versions of the OpenAI o1 model on several key metrics.
Sky-T1 stands out as the first truly open source reasoning model, as it can be replicated from scratch; The team not only published the weights of the model, but also the dataset used to train it and the code necessary for its training.
Let’s explore the main characteristics of this model, its comparison with other similar ones and what its designation as an open source model implies.
Main features of the Sky-T1-32B-Preview model
Sky-T1-32B-Preview is a language model designed to excel in specific domains such as mathematics and programming. Some of its key features include:
- Efficient design: NovaSky managed to train the model with a budget significantly lower than average for similarly sized models. This demonstrates an innovative approach to optimizing resources and training.
- Competitive performance: Although it is in preview, Sky-T1 has proven to be able to compete with more expensive models in tests related to mathematics and coding tasks.
- Access to model weights: NovaSky has made model weights available, making it easier for developers to experiment and build custom applications.
Unlike some competitors, Sky-T1 offers a complete set of resources, aligning with open source principles.
What does it mean for a model to be open source?
The Open Source Initiative (OSI) recently released version 1.0 of its Open Source Artificial Intelligence (OSAID) definition. This standard establishes the requirements that a model must meet to be considered truly open:
Full transparency
An open source model should provide detailed information about its design, including training data and the code used.
Access and modification
Developers should be free to use, modify, and redistribute the model without significant restrictions.
Provenance of data
It is necessary to disclose the origin and processing of the data used in training.
NovaSky ensures that the Sky-T1-32B-Preview meets these conditions, offering developers unprecedented access to the model’s key components.
However, this compliance must be evaluated under the standards established by the OSI to ensure that this is not a case of “open source in name only.”
Comparison with other models
The table below shows how the Sky-T1-32B-Preview stacks up against other popular models in several key aspects. Sky-T1 stands out by offering full access to data, code and model weights, making it attractive to developers and researchers.
In contrast, models like Journey and o1 have a more specialized approach to programming, but lack transparency in other aspects. This limits its adoption in projects that require full access to resources.
Using a variety of tests that assess math and coding skills, we’ll examine how Sky-T1 performs against competitors such as Qwen-2.5-32B-Instruct, QwQ, and o1-preview.
- Math500: Sky-T1-32B-Preview scored highly (82.4), second only to QwQ (85.4). This indicates his solid ability in solving mathematical problems.
- AIME2024: Sky-T1-32B-Preview showed a good performance (43.3), although QwQ led in this metric. Even so, Sky-T1 competes effectively with other models.
- LiveCodeBench-Easy: Sky-T1 performed well (86.3), although it was outperformed by o1-preview (92.9) and QwQ (90.7).
- LiveCodeBench-Medium: Sky-T1 led with a score of 56.8, outperforming QwQ and o1-preview, demonstrating its strength in medium-difficulty coding tasks.
- LiveCodeBench-Hard: Although all scores were low, Sky-T1 led with 17.9, showing its ability in difficult coding tasks.
- GPQA-Diamond: o1-preview dominated this test with 75.2, while Sky-T1 scored 56.8, showing good performance in generating high-quality answers.
Sky-T1-32B-Preview demonstrates solid and competitive performance in various tests, excelling especially in medium difficulty and difficult coding tasks and advanced mathematical problems.
How to test Sky-T1-32B-Preview?
NovaSky has made several ways available to the community to experiment with Sky-T1:
- Downloadable weights: Developers can access model weights from the NovaSky Github page. This allows the model to be implemented in its own infrastructure.
- Cloud environments: NovaSky has also left links to cloud platforms to test the model without the need for specialized hardware. For example, you can try Sky-T1 on HuggingFace.
- Complete documentation: The documentation includes code examples and practical applications, making the model easy to use even for beginner users.
These options reinforce NovaSky’s commitment to democratizing AI and promoting open practices.
The debate on open source in AI
The OSI definition of open source has generated controversy in the industry. Although Sky-T1 meets key requirements, other tech giants, such as Meta, have been criticized for labeling their models as “open source” despite imposing significant restrictions.
For example, Llama de Meta models require special licenses for large platforms, while Stability AI limits commercial use of its models.
These cases underscore the need for clear, enforceable standards to prevent abuse of the term “open source.” The OSI hopes that the AI community will adopt its definition as a reference, but there are still legal and technical challenges that could hinder its widespread application.
This post is also available in: