OstrichHermit/OH-Unity-GameDev-Skills / media-pipe-unity-skill
media-pipe-unity
VerifiedMediaPipe Unity Plugin 集成。用于计算机视觉任务、手部/面部/姿态追踪、手势识别、目标检测以及 Unity 中的 ML 解决方案。
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.1 KB |
| references/API-Overview.md | markdown | 25.4 KB |
| references/Advanced-Topics.md | markdown | 6.5 KB |
| references/FAQ.md | markdown | 2.5 KB |
| references/Getting-Started.md | markdown | 55.0 KB |
| references/Home.md | markdown | 716 B |
| references/Installation-Guide.md | markdown | 80 B |
| references/MediaPipeUnityPlugin-README.md | markdown | 15.9 KB |
| references/Tutorial-Hello-World.md | markdown | 9.1 KB |
| references/Tutorial-Task-API.md | markdown | 20.8 KB |
references/Tutorial-Hello-World.md
View on GitHub# Hello World!
> :bell: If you're new to MediaPipe, consider reading the [Framework Concepts](https://ai.google.dev/edge/mediapipe/framework/framework_concepts/overview) article first.
> :skull_and_crossbones: On Windows, some of the code below might cause UnityEditor to crash. Check [Technical Limitations](../README.md#warning-technical-limitations) for more information.
Let's write our first program!
> :bell: The following code is based on [mediapipe/examples/desktop/examples/hello_world/hello_world.cc](https://github.com/google/mediapipe/blob/cf101e62a9d49a51be76836b2b8e5ba5c06b5da0/mediapipe/examples/desktop/hello_world/hello_world.cc).
> :bell: You can find the complete code at [Tutorial/Hello World](https://github.com/homuler/MediaPipeUnityPlugin/tree/master/Assets/MediaPipeUnity/Tutorial/Hello%20World).
## Send the input
To use the Calculators provided by MediaPipe, we typically need to set up a [`CalculatorGraph`](https://ai.google.dev/edge/mediapipe/framework/framework_concepts/graphs). Let's start with that!
> :bell: Each `CalculatorGraph` requires its own configuration ([`CalculatorGraphConfig`](https://ai.google.dev/edge/mediapipe/framework/framework_concepts/graphs#graph_config)).
```cs
var configText = @"
input_stream: ""in""
output_stream: ""out""
node {
calculator: ""PassThroughCalculator""
input_stream: ""in""
output_stream: ""out1""
}
node {
calculator: ""PassThroughCalculator""
input_stream: ""out1""
output_stream: ""out""
}
";
var graph = new CalculatorGraph(configText);
```
To run a `CalculatorGraph`, call the `StartRun` method.
```cs
graph.StartRun();
```
Note that the `StartRun` method will throw an exception if there is an error.
Now that we've started the graph, let's provide some input to the `CalculatorGraph`.
Let's say we want to give a sequence of 10 strings (`"Hello World!"`) as input.
```cs
for (var i = 0; i < 10; i++)
{
// Send input to running graph
}
```
In MediaPipe, inputs are passed through a class called [`Packet`](https://ai.google.dev/edge/mediapipe/framework/framework_concepts/packets).
```cs
var input = Packet.CreateString("Hello World!");
```
To pass an input `Packet` to the `CalculatorGraph`, we use the `AddPacketToInputStream` method.\
In this example, our `CalculatorGraph` has a single input stream named `in`.
> :bell: It depends on the `CalculatorGraphConfig`. `CalculatorGraph` can have multiple input streams
```cs
for (var i = 0; i < 10; i++)
{
var input = Packet.CreateString("Hello World!");
graph.AddPacketToInputStream("in", input); // NOTE: Packet is disposed automatically
}
```
`CalculatorGraph#AddPacketToInputStream` may also throw an exception, for instance, when the input type is invalid.
When we're finished, we need to:
1. Close all input streams
2. Dispose of the `CalculatorGraph`
Here's how we do that:
```cs
using var graph = new CalculatorGraph(configText);
// ...
graph.CloseInputStream("in");
graph.WaitUntilDone();
```
For now, let's just run the code we've written so far.
Save the following code as `HelloWorld.cs`, attach it to an empty `GameObject` and play the scene.
```cs
using UnityEngine;
namespace Mediapipe.Unity.Tutorial
{
public class HelloWorld : MonoBehaviour
{
private void Start()
{
var configText = @"
input_stream: ""in""
output_stream: ""out""
node {
calculator: ""PassThroughCalculator""
input_stream: ""in""
output_stream: ""out1""
}
node {
calculator: ""PassThroughCalculator""
input_stream: ""out1""
output_stream: ""out""
}
";
using var graph = new CalculatorGraph(configText);
graph.StartRun();
for (var i = 0; i < 10; i++)
{
var input = Packet.CreateString("Hello World!");
graph.AddPacketToInputStream("in", input);
}
graph.CloseInputStream("in");
graph.WaitUntilDone();
Debug.Log("Done");
}
}
}
```

Oops, an error occurred.
```txt
BadStatusException: INVALID_ARGUMENT: Graph has errors:
; In stream "in", timestamp not specified or set to illegal value: Timestamp::Unset()
Mediapipe.Status.AssertOk () (at ./Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Port/Status.cs:168)
Mediapipe.MpResourceHandle.AssertStatusOk (System.IntPtr statusPtr) (at ./Packages/com.github.homuler.mediapipe/Runtime/Scripts/Core/MpResourceHandle.cs:115)
Mediapipe.CalculatorGraph.AddPacketToInputStream[T] (System.String streamName, Mediapipe.Packet`1[TValue] packet) (at ./Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/CalculatorGraph.cs:167)
Mediapipe.Unity.Tutorial.HelloWorld.Start () (at Assets/MediaPipeUnity/Tutorial/Hello World/HelloWorld.cs:35)
```
Each input packet [should have a timestamp](https://ai.google.dev/edge/mediapipe/framework/framework_concepts/packets), but it does not appear to be set.
Let's fix the code that initializes a `Packet` as follows:
```cs
// var input = Packet.CreateString("Hello World!");
var input = Packet.CreateStringAt("Hello World!", i);
```

This time it seems to be working.\
But wait, we are not receiving the `CalculatorGraph` output!
## Get the output
To receive output from the `CalculatorGraph`, we need to set up an output stream poller before running the graph.\
In this example, our graph has a single output stream named `out`.
> :bell: It depends on the `CalculatorGraphConfig`. `CalculatorGraph` can have multiple output streams.
```cs
using var graph = new CalculatorGraph(configText);
// Initialize an `OutputStreamPoller`. Note that the output type is string.
using var poller = graph.AddOutputStreamPoller<string>("out");
graph.StartRun();
```
Then, we can get output using the `OutputStreamPoller#Next`.\
Like inputs, outputs must be received through packets.
```cs
graph.CloseInputStream("in");
// Initialize an empty packet
using var output = new Packet<string>();
while (poller.Next(output))
{
Debug.Log(output.Get());
}
graph.WaitUntilDone();
```
Now, our code would look as follows.
Note that `OutputStreamPoller` and `Packet` also need to be disposed of.
```cs
using UnityEngine;
namespace Mediapipe.Unity.Tutorial
{
public class HelloWorld : MonoBehaviour
{
private void Start()
{
var configText = @"
input_stream: ""in""
output_stream: ""out""
node {
calculator: ""PassThroughCalculator""
input_stream: ""in""
output_stream: ""out1""
}
node {
calculator: ""PassThroughCalculator""
input_stream: ""out1""
output_stream: ""out""
}
";
using var graph = new CalculatorGraph(configText);
using var poller = graph.AddOutputStreamPoller<string>("out");
graph.StartRun();
for (var i = 0; i < 10; i++)
{
var input = Packet.CreateStringAt("Hello World!", i);
graph.AddPacketToInputStream("in", input);
}
graph.CloseInputStream("in");
using var output = new Packet<string>();
while (poller.Next(output))
{
Debug.Log(output.Get());
}
graph.WaitUntilDone();
Debug.Log("Done");
}
}
}
```

## Validate the config format
What happens if the config format is invalid?
```cs
using var graph = new CalculatorGraph("invalid format");
```

Hmm, the constructor fails, which is probably the behavior it should be.\
Let's check [`Editor.log`](https://docs.unity3d.com/Manual/LogFiles.html).
```txt
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:335] Error parsing text-format mediapipe.CalculatorGraphConfig: 1:9: Message type "mediapipe.CalculatorGraphConfig" has no field named "invalid".
MediaPipeException: Failed to parse config text. See error logs for more details
at Mediapipe.CalculatorGraphConfigExtension.ParseFromTextFormat (Google.Protobuf.MessageParser`1[T] _, System.String configText) [0x0001e] in ./Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/CalculatorGraphConfigExtension.cs:21
at Mediapipe.CalculatorGraph..ctor (System.String textFormatConfig) [0x00000] in ./Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/CalculatorGraph.cs:33
at Mediapipe.Unity.Tutorial.HelloWorld.Start () [0x00000] in /home/homuler/Development/unity/MediaPipeUnityPlugin/Assets/MediaPipeUnity/Tutorial/Hello World/HelloWorld.cs:29
```
Not too bad, but it's inconvenient to check `Editor.log` every time.\
Let's fix it so that the logs are visible in the Console Window.
```cs
Protobuf.SetLogHandler(Protobuf.DefaultLogHandler);
var graph = new CalculatorGraph("invalid format");
```

Great!\
However, there's an important detail to handle: we need to reset the log handler when the application exits to prevent potential crashes (SIGSEGV).\
Add this code to restore the default `LogHandler`:
```cs
void OnApplicationQuit()
{
Protobuf.ResetLogHandler();
}
```Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |