https://scicode-bench.github.io\n","updatedAt":"2024-07-22T07:59:02.430Z","author":{"_id":"6610c90c6c59fff43622eef7","avatarUrl":"/avatars/f832a3918101aa5f0834f1456b154c37.svg","fullname":"Yanxin Lu","name":"amber1120","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3557124435901642},"editors":["amber1120"],"editorAvatarUrls":["/avatars/f832a3918101aa5f0834f1456b154c37.svg"],"reactions":[],"isReport":false},"replies":[{"id":"66a24e137fe09e2a65d89b38","author":{"_id":"5f1158120c833276f61f1a84","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg","fullname":"Niels Rogge","name":"nielsr","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":975},"createdAt":"2024-07-25T13:07:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Hi @amber1120 congrats on this work!\n\nAre you planning to share the dataset on the hub? Here's a guide: https://huggingface.co/docs/datasets/loading.\n\nThe dataset could then be loaded in 2 lines of code, like so:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"your-hf-organization/scicode\")\n```\nIt could then also be linked to this paper, as explained here: https://huggingface.co/docs/hub/en/datasets-cards#linking-a-paper.\n\nWe could also set up a Space using Gradio for the leaderboard.\n\nLet me know if you need any help!\n\nCheers,\nNiels\nOpen-source @ HF","html":"
Hi \n\n@amber1120\n\t congrats on this work!
\nAre you planning to share the dataset on the hub? Here's a guide: https://huggingface.co/docs/datasets/loading.
\nThe dataset could then be loaded in 2 lines of code, like so:
\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"your-hf-organization/scicode\")\nIt could then also be linked to this paper, as explained here: https://huggingface.co/docs/hub/en/datasets-cards#linking-a-paper.
\nWe could also set up a Space using Gradio for the leaderboard.
\nLet me know if you need any help!
\nCheers,
Niels
Open-source @ HF
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024) \n
- PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models (2024) \n
- CodeRAG-Bench: Can Retrieval Augment Code Generation? (2024) \n
- AICoderEval: Improving AI Domain Code Generation of Large Language Models (2024) \n
- A Survey on Large Language Models for Code Generation (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2407.13168
SciCode: A Research Coding Benchmark Curated by Scientists
Published on Jul 18, 2024
Abstract
SciCode, a scientist-curated coding benchmark, evaluates language models' ability to solve scientific research problems across various fields, highlighting current capabilities and future challenges in AI-assisted science.
AI-generated summary
Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.
Community
Hi @amber1120 congrats on this work!
Are you planning to share the dataset on the hub? Here's a guide: https://huggingface.co/docs/datasets/loading.
The dataset could then be loaded in 2 lines of code, like so:
from datasets import load_dataset
dataset = load_dataset("your-hf-organization/scicode")
It could then also be linked to this paper, as explained here: https://huggingface.co/docs/hub/en/datasets-cards#linking-a-paper.
We could also set up a Space using Gradio for the leaderboard.
Let me know if you need any help!
Cheers,
Niels
Open-source @ HF
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024)
- PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models (2024)
- CodeRAG-Bench: Can Retrieval Augment Code Generation? (2024)
- AICoderEval: Improving AI Domain Code Generation of Large Language Models (2024)
- A Survey on Large Language Models for Code Generation (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2407.13168 in a model README.md to link it from this page.
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2407.13168 in a Space README.md to link it from this page.