It's not often I get to explore, working on something open-ended where there should be a way to do it but it's not clear where to start. First you have to find out if you can do it, and then figure out how. When I see the opportunity for that work, I jump on it. I love exploring.
But it can be overwhelming if you're accustomed to having all the information and a clear goal before you start, so here are a few things I've found helpful.
It's tempting to hit the ground running, but pause and set up a few things before you start. You'll thank yourself later. I like to open a new browser window and a blank Google doc to collect all my tabs for this work in one place.
It's helpful to prepare your mind too: get into an exploratory mindset where you're comfortable not having all the information. There will be times when you'll feel lost, be okay with that.
Look for anything you think could help you find your way. Start with internal docs and Google. Search Slack and GitHub to see if anyone has tried to do something similar. You're not looking for something you can copy/paste, but sometimes concrete examples can be more helpful than abstract explanations.
Take very rough notes at first. These are just for you, so they can be disorganized. Jot down thoughts as you come across anything that might be helpful, questions that come up, links, etc. These are breadcrumbs for you, don't worry about cleaning them up.
You will. The path won't always be obvious. The whole point of this is that it's uncharted territory, at least for you. Documentation will be sparse or missing. You'll get into a situation where something's not working but it's hard to diagnose what's wrong.
When this happens, take a break. You won't want to, you'll sense the solution is just around the corner. And it probably is, but you'll turn that corner a lot faster after getting some space. Maybe stop for the day. It's amazing how much clarity comes from stepping away from something.
Now's a good time to look back at your notes. Write down your findings so far, or the state of what you know: "I can see A and B are definitely doing the right thing, so it's either C or D that's still broken." The act of writing it down helps you keep focused.
When something goes right, or when you find a missing piece that clarifies something, celebrate! Take a break, tell someone, add it to your notes. As the breakthroughs add up it's easy to forget how the small things started to snowball.
If you find incomplete documentation during your travels, fill in the gaps! You can include these notes in a writeup, or even better: if it's an open source project propose additions to their docs. This can be intimidating, but even if you get something wrong you'll open a conversation with the maintainers. They usually want things to be as clear as possible (it means they'll spend less time answering questions) so they have an incentive to work with you to get it right.
Hey, great work! You figured out how to tackle a vague problem and learned something along the way. Now to spread that knowledge. What will others need to know? Anything surprising? Any blockers to moving forward? Usually the outcome of an exploration is a conclusion you or others can act on. Write up the takeaway in a few bullet points, condensing your findings into something easily digestible for teammates who are still missing the context you just gained.
It's a jungle out there, I hope next time this guide will help!