OpenAI has introduced a new model called CriticGPT, which is based on GPT-4. Unlike other models meant for consumers, CriticGPT is designed to critique ChatGPT responses, helping human trainers identify mistakes during reinforcement learning from human feedback (RLHF).

CriticGPT is based on GPT-4 and aims to help human trainers at OpenAI catch errors in ChatGPT’s code output. OpenAI claims that code reviewed by CriticGPT can outperform unreviewed code by 60%. 

The company is currently integrating CriticGPT-like models into the RLHF labeling pipeline to assist AI trainers in evaluating outputs from advanced AI systems.

According to OpenAI, models like CriticGPT can improve the accuracy of ChatGPT. Further, the new model can identify errors that humans might miss as the models become more knowledgeable.

The training process for CriticGPT involved manually editing ChatGPT-generated code, introducing new errors into the code along with sample feedback to train the model to identify common and less common mistakes easily.

Similar to human suggestions, CriticGPT’s suggestions are not always correct. However, combining human input with CriticGPT’s feedback is said to outperform unassisted human trainers and helps trainers write comprehensive critiques while producing fewer errors.

OpenAI also stated that CriticGPT might spread real-world mistakes across many parts of the answer and cannot evaluate an extremely complex task or response.

Overall, this new AI model is expected to help human trainers produce better RLHF data for GPT-4, and OpenAI plans to further scale this work.