after +10 years in research, this is how I organize running (AI)...
Abdullah Hamdi@Eng_Hemdi
74 views
Sep 04, 2026
~2 min read
Advertisement
1
after +10 years in research, this is how I organize running (AI) research projects. Usually researchers do not document this in research papers. Its the "truth-seeking" aspect of doing research. The journey not the outcome.
Before delving into deep R&D cycles, there need to be a "gut-feeling" of whats happening, this "gut-feeling" need to develop with quick interaction with the recent tech and some interesting positive or negative scenarios that tells you to spend some time on this phenomena.
at the beginning of project , I organize experiments with experiments sets (00 to 99 ) each round of experiments has its own ordered ID where a global dictionary describes that round of tests , in each set dir I can run up to millions of experiments with random gnerated ID ( something like GLE876) those can be millions dirs each with a config json file and a results json, this to account for random noise and to test different config changes of that round at once. Any artifacts (visuals, text, meta info ) generated should be also saved in each exp subdir along with the config and results json. Any major change during development should be a new set of exps. No more than 3 ideas to test each round at once.
after aggregating the all of exp dirs results json each round and plotting a summary, it should ideally give a hint of how things work (whats true and whats not), whats working/hurting (>10% improvement or >10% degradation) and whats just noise (~ +/- 1%), this "hint" should be logged on same file describing the set above.
each iteration code/setup should be committed to github with ID so a fallback to older ID version of the research project is possible.
ideally a small set of indicators need to be tracked , not just numbers (metrics) but also maybe visuals, complexity (inverse) and cost (compute,memory, money). The more the indicators to track in a research project, the worst the expected research outcome.
At end of project, "truth-seeking" is over, and "story-telling" kicks in. A new ablations/ ID ordered sets emerge each numbered also from 00 to 99 to justify the choices along the way. Basically preparing to publish the results with 00 being the best setup reached after all iterations.
Most research projects will never exceed 100 iterations or the complexity exceeds 100 ablations (design choices ). If it does there is something wrong and the project need to be either finished or killed.
I hope this can help someone ( or some AI agent😃 ) trying to do research without getting lost.
Before delving into deep R&D cycles, there need to be a "gut-feeling" of whats happening, this "gut-feeling" need to develop with quick interaction with the recent tech and some interesting positive or negative scenarios that tells you to spend some time on this phenomena.
at the beginning of project , I organize experiments with experiments sets (00 to 99 ) each round of experiments has its own ordered ID where a global dictionary describes that round of tests , in each set dir I can run up to millions of experiments with random gnerated ID ( something like GLE876) those can be millions dirs each with a config json file and a results json, this to account for random noise and to test different config changes of that round at once. Any artifacts (visuals, text, meta info ) generated should be also saved in each exp subdir along with the config and results json. Any major change during development should be a new set of exps. No more than 3 ideas to test each round at once.
after aggregating the all of exp dirs results json each round and plotting a summary, it should ideally give a hint of how things work (whats true and whats not), whats working/hurting (>10% improvement or >10% degradation) and whats just noise (~ +/- 1%), this "hint" should be logged on same file describing the set above.
each iteration code/setup should be committed to github with ID so a fallback to older ID version of the research project is possible.
ideally a small set of indicators need to be tracked , not just numbers (metrics) but also maybe visuals, complexity (inverse) and cost (compute,memory, money). The more the indicators to track in a research project, the worst the expected research outcome.
At end of project, "truth-seeking" is over, and "story-telling" kicks in. A new ablations/ ID ordered sets emerge each numbered also from 00 to 99 to justify the choices along the way. Basically preparing to publish the results with 00 being the best setup reached after all iterations.
Most research projects will never exceed 100 iterations or the complexity exceeds 100 ablations (design choices ). If it does there is something wrong and the project need to be either finished or killed.
I hope this can help someone ( or some AI agent😃 ) trying to do research without getting lost.
