Viraat D.

Nvidia's new open-weight model, Nemotron-3-Nano, takes the medal for most transparent model from a commercial company this year. Scoring a 78% score on Trustible's Model Ratings, it ties with Olmo-2 and really shines when it comes to Data Transparency. Highlights: 📕 Nearly Open-Data: Nemo is trained almost entirely on open-data. The developers go a step further and document WHEN each external dataset was documented which can be important for reproducibility, if data changes over time. However, the documentation lists out several private datasets - so it isn't 100% open-source ⚒️ Training Details: The Technical Report explains the training process with detailed hyper-parameters, improving the accessibility for other developers who want to build similar systems 🤩 Public Evaluation Code: For the evaluations that were performed, Nvidia links to public code - this is an increasingly rare practice. Even open-weight Llama-3 did not document how they calculated their benchmark scores. Lowlights: 📉 Data vs Evaluation trade-off: Similar to other models with high data transparency, Nemo lags on evaluations. They perform no evaluations of limitations or adversarial testing. While our ratings don't require extensive red-teaming, we expect a discussion of this topic to help the end-user make informed decisions. 🔎 Documentation Organization: This rating took us a while to create, because several important details were not front and center. While I liked the detailed list of datasets, Ethics and Privacy details should be highlighted earlier in the documents. Using a Model Cards type format would have raised the Documentation Support rating from Medium to High. Detailed Rating: https://lnkd.in/e6MVXzfv

25

1 Comment

Northflank is partnering again with Cerebral Valley, PyTorch, CoreWeave (and more) for the OpenEnv Hackathon in San Francisco, March 7–8. Participants will be building RL environments and post-training base models across 5 themes, solving the most pressing problems in RL and agentic orchestration. We'll be supporting teams in deploying their models through the Northflank platform with compute from our wonderful partners, CoreWeave (Matthew Lu, Jacob Feldman). $100K+ prize pool and and a stacked lineup of judges and mentors from Meta, Hugging Face, University of California, Berkeley, Cursor, Scale AI, and more. You can submit an application at the link in comments.

56

3 Comments