diff --git a/website/blog/2022-05-20-preparing-for-headless-flipper.md b/website/blog/2022-05-20-preparing-for-headless-flipper.md new file mode 100644 index 000000000..b7e0cd537 --- /dev/null +++ b/website/blog/2022-05-20-preparing-for-headless-flipper.md @@ -0,0 +1,337 @@ +--- +title: Headless Flipper - what it means for plugin developers +author: Andrey Goncharov +author_title: Software Engineer +author_url: https://github.com/aigoncharov +author_image_url: https://github.com/aigoncharov.png +tags: [flipper, headless, plugins] +description: + Flipper is moving from an Electron monolith to a headless Node.js app with a + web UI. It reshapes how we think about plugins and what plugins can do. We + talk about what changes and how to prepare our plugins for the migration. +image: /img/preparing-for-headless-flipper.jpg +hide_table_of_contents: false +--- + +![Cover image](/img/preparing-for-headless-flipper.jpg) + +We know Flipper as an Electron desktop app that serves mobile developers as +their debugging companion. Thousands of people use Flipper every day to tinker +with their app and get to the bottom of tricky problems. + +As announced in the previous +[roadmap post](https://fbflipper.com/blog/2021/10/14/roadmap/), we are committed +to amplifying how Flipper could improve the quality of our software. We want +take Flipper beyond its current role as a complementary debugging tool, provide +a powerful API, and allow using Flipper in more than just the GUI context (we +call it "headless mode"). Imagine talking to your mobile device (or anything +else that runs Flipper Client) from your terminal. Imagine deploying Flipper +remotely in the cloud and interacting with it from your laptop. Imagine using +your favorite plugins for automated testing. + +In this post we cover: + +- How Flipper changes to facilitate the headless mode +- How it affects plugins +- A migration guide + + + +## How Flipper changes + +Let us take a look at how it works today as an Electron app. + +![Flipper Electron architecture](/img/flipper-arch-electron.jpg) + +Here is what happens: + +1. Flipper starts as an Electron application. + 1. WebSocket server starts. + 2. Device discovery starts via adb/idb/metro. + 3. Electron shows a web view with Flipper UI (React). + 4. Flipper UI queries the device discovery service for a list of devices. +2. At this point, Flipper can already run + ["device" plugins](https://fbflipper.com/docs/extending/desktop-plugin-structure/#creating-a-device-plugin). + These plugins do not receive a connection to a running app. They talk to the + device via adb/idb/metro. +3. An app starts on the device. +4. Flipper Client embedded in the app connects to the WebSocket server. +5. Flipper updates the list of known clients and reflects it in the UI. +6. Now Flipper can run + ["client" plugins](https://fbflipper.com/docs/extending/desktop-plugin-structure/#creating-a-client-plugin). +7. Client plugins talk to the device application over the WebSocket connection. + +> You can start Flipper Electron with `yarn start` from the `/desktop` folder. + +Here is how Flipper Headless works. + +![Flipper Headless architecture](/img/flipper-arch-headless.jpg) + +1. Flipper starts as a Node.js application. + 1. WebSocket server starts. + 2. Device discovery starts via adb/idb/metro. + 3. Web server starts. + 4. It serves Flipper UI to a browser. + 5. Flipper UI connects to the WebSocket server. + 6. Flipper UI sends a message over the WebSocket connection to query the + device discovery service for a list of devices. +2. At this point, Flipper can already run + ["device" plugins](https://fbflipper.com/docs/extending/desktop-plugin-structure/#creating-a-device-plugin). + These plugins do not receive a connection to a running app. They talk to the + device via adb/idb/metro. +3. An app starts on the device. +4. Flipper Client embedded in the app connects to the WebSocket server. +5. Flipper updates the list of known clients. It sends a message over the + WebSocket connection to Flipper UI with the information about the new device. +6. Now Flipper can run + ["client" plugins](https://fbflipper.com/docs/extending/desktop-plugin-structure/#creating-a-client-plugin). +7. Client plugins talk to the device application over the WebSocket bridge - the + connection from Flipper UI to Flipper WebSocket server piped to the + connection from the device application to the Flipper WebSocket server. + +> You can start Flipper Electron with `yarn flipper-server` from the `/desktop` +> folder. + +## How it affects plugins + +Plugins are hosted by Flipper UI. When it was a part of the Electron app, there +was no problem. Plugins could access any Node.js APIs thanks to Electron magic. +There were no constraints on what plugins could do. After making Flipper UI a +proper web app running in a browser, we limited what plugins can do. They no +longer can access the network stack or the file system because there are no +corresponding browser APIs. Yet, we want to keep Flipper flexible and provide as +much freedom to plugin developers as possible. Moreover, we could not leave the +existing plugins without a clear migration path. + +![Flipper remote Node.js API](/img/flipper-node-apis.jpg) + +Since we already have a WebSocket connection between Flipper UI and Flipper +Server, we can model almost any request-response and even stream-based Node.js +APIs over it. So far, we exposed a curated subset of them: + +- child_process + - exec +- fs (and [fs-extra](https://www.npmjs.com/package/fs-extra)) + - constants + - access + - pathExists + - unlink + - mkdir + - rm + - copyFile + - stat + - readlink + - readFile + - writeFile + +We also provided a way to +[download a file](https://github.com/facebook/flipper/blob/0f038218f893d86e91714cd91eed8e37d756386c/desktop/flipper-plugin/src/plugin/FlipperLib.tsx#L83) +or send requests to the +[internal infrastructure](https://github.com/facebook/flipper/blob/0f038218f893d86e91714cd91eed8e37d756386c/desktop/flipper-plugin/src/plugin/FlipperLib.tsx#L186). + +> Please, find the complete list of available APIs on +> [GitHub](https://github.com/facebook/flipper/blob/0f038218f893d86e91714cd91eed8e37d756386c/desktop/flipper-plugin/src/plugin/FlipperLib.tsx#L95). +> [Here are Node.js API abstractions](https://github.com/facebook/flipper/blob/0f038218f893d86e91714cd91eed8e37d756386c/desktop/flipper-plugin/src/plugin/FlipperLib.tsx#L47) +> specifically. + +As you might have noticed, all exposed APIs are of the request-response nature. +They assume a short finite controlled lifespan. Yet, some plugins start +additional web servers or spawn long-living child processes. To control their +lifetime we need to track them on Flipper Server side and stop them whenever +Flipper UI disconnects. Say hello to a new experimental feature - Flipper Server +Add-ons! + +![Flipper Server Add-on](/img/flipper-add-on.jpg) + +Now, every flipper plugin could have "server add-on" meta-information. Whenever +a Flipper plugin that has a corresponding Server Add-on starts, it sends a +command to Flipper Server to start its Server Add-on counterpart. Flipper plugin +that lives in a browser inside of Flipper UI talks to its server add-on over the +WebSocket connection. Whenever a user disables a plugin, Flipper sends a command +to Flipper Server to stop the add-on. At the same time, if Flipper UI crashes or +the user just closes the tab, Flipper Server can kill the server add-on on its +own. + +Flipper plugin can talk to its server add-on companion (see +`onServerAddOnMessage`, `onServerAddOnUnhandledMessage`, `sendToServerAddOn` in +[the docs](https://fbflipper.com/docs/extending/flipper-plugin/#pluginclient)) +and act whenever it starts or stops (see `onServerAddOnStart`, +`onServerAddOnStop` in +[the docs](https://fbflipper.com/docs/extending/flipper-plugin/#pluginclient)). + +Say, you wrote an ultimate library to find primes. You were cautious of the +resource consumption, so you did it in Rust. You created a CLI interface for +your new shiny library. Now, you want your Flipper plugin to use it. It takes a +long time to find a prime and you want to keep track of the progress. You could +use `getFlipperLib().remoteServerContext.childProcess.exec`, but it is not +flexible enough to monitor progress updates that your CLI sends to stdout. Here +is how you could approach it: + +```tsx +// contract.tsx +export interface ServerAddOnEvents { + // Server add-on sends "progress" events with the progress updates + progress: number; +} +export interface ServerAddOnMethods { + // Client plugin send "findPrime" messages to the server add-on to start finding primes + findPrime: () => Promise; +} + +// index.tsx (your plugin) +import {usePlugin, useValue, createState, PluginClient} from 'flipper-plugin'; +import {ServerAddOnEvents, ServerAddOnMethods} from './contract'; + +export const plugin = ( + client: PluginClient<{}, {}, ServerAddOnEvents, ServerAddOnMethods>, +) => { + const prime = createState(null); + const progress = createState(0); + + client.onServerAddOnStart(async () => { + const newPrime = await client.sendToServerAddOn('findPrime'); + prime.set(newPrime); + }); + + client.onServerAddOnStart(() => { + client.onServerAddOnMessage('progress', progressUpdate => { + progress.set(progressUpdate); + }); + }); + + return { + prime, + progress, + }; +}; + +export const Component = () => { + const pluginInstance = usePlugin(plugin); + const prime = useValue(pluginInstance.prime); + const progress = useValue(pluginInstance.progress); + + return
{prime ?? `Calculating (${progress}%) done...`}
; +}; + +// serverAddOn.tsx +import {ServerAddOn} from 'flipper-plugin'; +import {exec, ChildProcess} from 'child_process'; +import {ServerAddOnEvents, ServerAddOnMethods} from './contract'; + +const serverAddOn: ServerAddOn = + async connection => { + let findPrimeChildProcess: ChildProcess | undefined; + + connection.receive('findPrime', () => { + if (findPrimeChildProcess) { + // Allow only one findPrime request at a time. Finding primes is expensive! + throw new Error('Too many requests!'); + } + + // Start our awesome Rust lib + findPrimeChildProcess = exec('/find-prime-cli', {shell: true}); + + // Return a Promise that resolves when a prime is found. + // Flipper will serialize the value the promise is resolved with and send it oer the wire. + return new Promise(resolve => { + // Listen to stdout of the lib for the progress updates and, eventually, the prime + findPrimeChildProcess.stdout.on('data', data => { + // Say, data is a stringified JSON + const parsed = JSON.parse(data); + + if (parsed.type === 'progress') { + connection.send('progress', parsed.value); + return; + } + + // Allow new requests to find new primes + findPrimeChildProcess = undefined; + // If it is not a progress update, then a prime is found. + resolve(parsed.value); + }); + }); + }); + }; + +export default serverAddOn; +``` + +## Migration guide + +1. Examine your plugins for Node.js APIs. Replace them with + `getFlipperLib().remoteServerContext.*` calls. + + ```tsx + // before + import {mkdir} from 'fs/promises'; + + export const plugin = () => { + const myAwesomeFn = async () => { + await mkdir('/universe/dagobah'); + }; + + return { + myAwesomeFn, + }; + }; + + // after + import {getFlipperLib} from 'flipper-plugin'; + + export const plugin = () => { + const myAwesomeFn = async () => { + await getFlipperLib().remoteServerContext.mkdir('/universe/dagobah'); + }; + + return { + myAwesomeFn, + }; + }; + ``` + +2. If your plugin uses network stack of spawns a subprocess, consider creating a + Server Add-on. + + 1. In your plugin's folder create a new file - `serverAddOn.tsx` + 2. In your plugin's package.json add fields `serverAddOn` and + `flipperBundlerEntryServerAddOn` + + ```js + { + // ... + "serverAddOn": "dist/serverAddOn.js", + "flipperBundlerEntryServerAddOn": "serverAddOn.tsx", + // ... + } + ``` + + 3. Move your Node.js API calls to `serverAddOn.tsx` + +3. Verify your plugin works in a browser environment. + 1. Clone [Flipper repo](https://github.com/facebook/flipper). + 2. Navigate to the `desktop` folder. + 3. In your terminal run `yarn`. + 4. Run `yarn flipper-server`. + 5. Load your plugin and make sure it works. + +## P.S. Flipper needs you! + +Flipper is maintained by a small team at Meta, yet is serving over a hundred +plugins and dozens of different targets. Our team's goal is to support Flipper +as a plugin-based platform for which we maintain the infrastructure. We don't +typically invest in individual plugins, but we do love plugin improvements. + +For that reason, we've marked many requests in the issue tracker as +[PR Welcome](https://github.com/facebook/flipper/issues?q=is%3Aissue+is%3Aopen+label%3A%22PR+welcome%22). +Contributing changes should be as simple as cloning the +[repository](https://github.com/facebook/flipper) and running +`yarn && yarn start` in the `desktop/` folder. + +Investing in debugging tools, both generic ones or just for specific apps, will +benefit iteration speed. And we hope Flipper will make it as hassle free as +possible to create your debugging tools. For an overview of Flipper for React +Native, and why and how to build your own plugins, we recommend checking out the +[Flipper: The Extensible DevTool Platform for React Native](https://youtu.be/WltZTn3ODW4) +talk. + +Happy debugging! diff --git a/website/static/img/flipper-add-on.jpg b/website/static/img/flipper-add-on.jpg new file mode 100644 index 000000000..91b9d5055 Binary files /dev/null and b/website/static/img/flipper-add-on.jpg differ diff --git a/website/static/img/flipper-arch-electron.jpg b/website/static/img/flipper-arch-electron.jpg new file mode 100644 index 000000000..7ea9a0985 Binary files /dev/null and b/website/static/img/flipper-arch-electron.jpg differ diff --git a/website/static/img/flipper-arch-headless.jpg b/website/static/img/flipper-arch-headless.jpg new file mode 100644 index 000000000..67888eb54 Binary files /dev/null and b/website/static/img/flipper-arch-headless.jpg differ diff --git a/website/static/img/flipper-node-apis.jpg b/website/static/img/flipper-node-apis.jpg new file mode 100644 index 000000000..7a8d352d1 Binary files /dev/null and b/website/static/img/flipper-node-apis.jpg differ diff --git a/website/static/img/preparing-for-headless-flipper.jpg b/website/static/img/preparing-for-headless-flipper.jpg new file mode 100644 index 000000000..68bf06da3 Binary files /dev/null and b/website/static/img/preparing-for-headless-flipper.jpg differ