# Idea: Library that involves drawing and sound

**URL:** <https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790>\
**Category:** Project Guidance\
**Created:** [September 4, 2019, 6:58am UTC](https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790 "2019-09-04T06:58:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tanhengyeow](https://avatars.discourse-cdn.com/v4/letter/t/2bfe46/32.png) [@tanhengyeow](https://discourse.processing.org/u/tanhengyeow)\
**Post date:** [September 4, 2019, 6:58am UTC](https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790/1 "2019-09-04T06:58:53Z")

</div>

Hi all!

My name is Heng Yeow and I’m an open-source enthusiast. I’m interested in K12 education and developing creative learning experiences.

I’m thinking of supporting a p5 library that combines audio and drawing on canvas. Similar function calls that draw something would be mapped to the same audio, helping students to identify similar function patterns through the use of sound.

Would like to seek the community’s thoughts and feedback about it. Thank you 😃

---

<div class="post-metadata">

**Author:** ![jeremydouglass](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/jeremydouglass/32/20_2.png) [@jeremydouglass](https://discourse.processing.org/u/jeremydouglass)\
**Post date:** [September 4, 2019, 8:07am UTC](https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790/2 "2019-09-04T08:07:06Z")

</div>

> [@tanhengyeow](#):
>
> Similar function calls that draw something would be mapped to the same audio, helping students to identify similar function patterns through the use of sound.

Do you mean, for example, that rect and ellipse might produce different oscillator pitches? Or that 3x3 rects would be louder than one rect?

---

<div class="post-metadata">

**Author:** ![tanhengyeow](https://avatars.discourse-cdn.com/v4/letter/t/2bfe46/32.png) [@tanhengyeow](https://discourse.processing.org/u/tanhengyeow)\
**Post date:** [September 4, 2019, 9:50am UTC](https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790/3 "2019-09-04T09:50:44Z")

</div>

To put this into analogy, consider a tree. A tree 🌲 is made up of leaves, branches, a tree trunk etc. When drawing each part separately, the audio played will be different. This example could help students learn that a program is made up of multiple function calls intuitively based on different sounds heard.

Some thoughts I have - Is this something feasible and useful for p5? Would a p5 library be suitable for this type of concept? Are there other similar things out there?

I’m not that familiar with developing for the p5 ecosystem now but I’ll be willing to pick up the necessary skills 😄 Let me know what you think!

---

<div class="post-metadata">

**Author:** ![jeremydouglass](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/jeremydouglass/32/20_2.png) [@jeremydouglass](https://discourse.processing.org/u/jeremydouglass)\
**Post date:** [September 4, 2019, 5:08pm UTC](https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790/4 "2019-09-04T17:08:15Z")

</div>

> [@tanhengyeow](#):
>
> When drawing each part separately

This seems to be _the_ key part of your concept. By default, a collection of instructions in Processing / p5.js all apply to a single frame (~1/60th sec) – there is no automatic temporal dimension / animation the way that there is in, say, Logo, and no built-in heirarchy in the way of say, Context Free.

So, this tree-like random windmill code describes a trunk, branches, and leaves which evolve randomly. However, it draws _every_ element _every_ frame by default, so all the associated sounds would play all at once, at 60fps.

> **[p5.js Web Editor](https://editor.p5js.org/jeremydouglass/sketches/4LTEH7w0C)**
>
> A web editor for p5.js, a JavaScript library with the goal of making coding accessible to artists, designers, educators, and beginners.

So in order to align sounds with visual instructions in a sensible way – that animates a tree, with different tones for trunk, branches, leaves – you would need some kind of framework to infer that animation, otherwise the sketches themselves will be quite complex, and not appropriate for learners.

Depending on your animation framework concepts, that would give you the sound attachment points. So, if you were doing recursive descent and trying to animate a tree, you need to know whether to play each branch and its leaves (bang ding ding, bang ding ding) or every branch and then every leaf (bang bang, ding ding ding ding).

It is an interesting problem! Would love to hear more ideas about it.

---

<div class="post-metadata">

**Author:** ![jeremydouglass](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/jeremydouglass/32/20_2.png) [@jeremydouglass](https://discourse.processing.org/u/jeremydouglass)\
**Post date:** [October 25, 2019, 2:57am UTC](https://discourse.processing.org/t/idea-library-that-involves-drawing-and-sound/13790/5 "2019-10-25T02:57:49Z")

</div>

Just saw this, looks interesting:

- Joo, Woohun. “Sonifyd: A graphical approach for sound synthesis and synesthetic visual expression.” Georgia Institute of Technology, 2019. [https://smartech.gatech.edu/handle/1853/61519](https://smartech.gatech.edu/handle/1853/61519)