Transforming audio signal into midi signal

[elm]

Hi.

I'm new here and to audio programming. I have many years of programming but no noteworthy experience with audio programming.

I want to write a software that has an audio signal as input and should output a midi signal in realtime. The audio input is taken from a percussion instrument and the job is to output midi signals, when the percussion instrument is hit. For the beginning only the volume of the audio signal should be taken into account for the midi output. Later other characteristics.

My questions:
- Hard to do this?
- What programing language should I use? Requirements:
-- Open source
-- Linux (cross platform would be nice)
-- I'd prefer statically typed languages
-- I would love to use a functional programming language
-- Good IDE would be great (I love ReSharper for C# development)
- How can I go the first steps? Are there libraries to analyze audio streams in realtime or do you know tutorials or can you give me a short explanation? I have no idea, how those kind of data is handled.

Existing solutions would be an option, too. But if it's not too hard, I'd love to write it by myself to be able to play around with that, have the full control and learn.

I think, there are thousands of ways to do this and also linux is not the system number one for most of the sound programmers. But ... maybe you can help me or just tell my how you would start doing this (on a linux machine).

Thank you for reading

elm

[blue hell]

Welcome elm.

Maybe have a look at PD and ChucK, both open source and cross platform audio programming environments.

The ChucK audio related source code is based on the Synthesis Tool Kit (STK, also available on it's own) and it is pretty readable all - it may give you some ideas about how to deal with Audio / MIDI .. in case you really want to go DIY.

[JovianPyx]

If this is for live performance:

If the system is for percussion only, you might consider drum heads that are designed for this purpose - I'm sure you've seen them, they look like plastic drum heads with no drum resonator under them. They emit a signal that goes into a box and out comes MIDI. Each drum head can then output a signal that can be routed to a particular MIDI drum synth sound. As such, any of these heads can be assigned snare, tom, ride, crash, hat or whatever you like.

To do this DIY, people have used piezo-electric transducers (these are a kind of microphone) attached to real drums. The problem here is that when you strike the drum, you will get several/many pulses. The pulses can be high voltage, so your circuitry needs to deal with those high voltages and make logic friendly voltages out of them (so as not to blow up the microprocessor). And at the same time, the amplitude of the hit must be measured by an ADC (which also needs protection from high voltages). The signals then go into a microprocessor that "listens" for the drum hit pulses and when received the microprocessor sends out a MIDI message constructed for your purpose (MIDI note on messages have information about pitch, and velocity (striking force). The coding for this should be easily done in a device like a PIC or even an AVR (arduino) if the number of drums is small (like 4 to 8 - the main limitation will be the number of pins the microprocessor has). All of this stuff is handled by "magic" in a manufactured MIDI drum set.
--------------------------------------------------------------------------------------------------------------------------------

If the idea is to take a pre-recorded or mic'd drum track and convert the drum hits to MIDI, this will be a bit more complex and will require knowledge of DSP. The reason is that drum signals are quite dirty and your program will need to recognize the start of each drum hit. For example, if a cymbal is still ringing and a snare hit happens - how do you detect that there were two hits algorithmically and which drums were hit? Separation by pitch is one possible way, but not without it's flaws since drums contain a large amount of wide band noise. A mixed track would be the most difficult to manage.
--------------------------------------------------------------------------------------------------------------------------------

In my view, the operating system doesn't matter and depending on exactly what you want to do there may not be a need for an OS at all. That said, anything that can be done in Windows can be done with Linux. For Linux, you will need a rather deep understanding of ALSA to be able to do real-time sampling of an audio signal.

My language of choice would be C (or assembly language) since it's easy to control what can be annoying such as function call overhead and C has no garbage collection. I would steer clear of Java and C# because both use a runtime (Java being worse than C#) and both cause random garbage collection which will introduce random amounts of latency. Interpreted languages should be avoided. If this idea is for converting pre-recorded or mic'd drums, the biggest problem you will have is processing the mixed audio drum track to determine when a drum has been hit, what kind of drum it is and how hard it was hit. Lots of research has been done on this subject and there's no easy answer while at the same time the results are certainly not perfect. Problems include latency (late output drum timing) which may be worse where drum sounds are more layered. Drum type detection will also require lots of work and knowledge of DSP. If you want this for performance, I'd look into manufactured MIDI drum heads designed for this purpose - sure, they are a bit expensive, but they work.

[elm]

Uh, thanks for that replies!

To make it clear a bit: It's for live performance. Not for a drum kit but for something like a cajon (played with two hands - the food can be neglected). I wanted to start using an ordinary microphone, that I already have and if that works, I wanted to try to use some piezos to be (maybe) able to detect on which side of the cajon the hit has been made (which hand) and maybe also the vertical position. Buying something like a cajon with midi-out is not an option for me. My focus is on experimenting to build myself my own hybrid acoustic-electric-percussion-box.

That all must not be very precise, I think. But a low latency would be important.

I have never programmed microcontrollers. What would be the advantages over using an audio interface and programming the thing in the laptop?

Thanks a lot!

[JovianPyx]

Ah, thanks, let me see if I understand what you want to do.

I'm assuming one cajon instrument. A microphone might work, but placement may be critical. Perhaps a contact microphone would be best. FFT can give you information about pitch and amplitude, but there's many different ways to do it each with it's own personality such as faster, higher resolution, precision. To do any sort of determination of "what MIDI message should I send for this hit?", you will need DSP, whether FFT or something like a vocoder with multiple filters. Much also depends on what you want to express about the cajon hits inside the MIDI messages - or what will the MIDI messages be used for?

[elm]

After thinking more about what I want, I think, that MIDI is not the right choice.

Right now I'm traveling around and don't want to build or buy something. So my test setup would be: I have a guitar. The strings are muted or, for better understanding let's say, there are no strings on the guitar. I have a laptop with a microphone and headphones. My goal is this: When I beat the body(?) of the guitar with my fingers, I would like to hear something in my headphones, that would sound like if I would have beaten the "G" of a marimba. It is not important, that it sounds lige a marimba, but it is important that it sounds like a "G". When I change my code or hit a key or tip a food pedal, it should sound like an "A", when I beat the guitar again.

The "G" should be mixed into the more or less toneless sound of me beating the guitar. So if I would use my fingernails the "G" should sound different from a beat with the fingertip or with the palm.

Is this understandable? And do you think, this is possible?

[JovianPyx]

I read your first post again and saw something I didn't comment on properly. You mention Linux as an OS. I've recently had some excellent results using Linux on a Raspberry Pi 2 (and also 3). I did the DSP work within an ALSA framework. I have 24 bit audio cards on both. After the audio board is plugged into the Pi, the unit is about the size of a pack of cigarettes (so very portable). Pi costs $35(american) and the audio board costs $45. I was able to get 32 voices of karplus-strong, so you can see it's a very powerful device. It would easily do pass-through DSP effects like filters. Not sure if it's an electric or acoustic guitar, but the sound of thumping a guitar body is going to be fairly broadband and pitchless. However, if you passed this through a bandpass filter, you could add or an enhance a pitch. You could do this digitally with the equipment I have. You could also do this with an analog filter, but you would have to build that on perf-board or vector or strip-board. The analog way would be much less expensive. I can't say if it will work as well as you need/want.

I see that you said you didn't want to buy stuff, but I put the stuff about the filter just in case. I designed a digital synth that hits a filter with a pulse and I get a tonal thump. The synth plays a standard chromatic scale. I also wrote a tonal noise synth - a filter again - through which noise is passed and the filter selects a narrow band of frequencies to make a shhhhh sound that has a tone. It also plays a chromatic scale. So I can say that using a filter to add pitch to something that is unpitched works. The filter I used in all cases is called State Variable Filter.