System Identification

For Q2, 63% isn't randomly chosen. It has a very rich meaning.

Mathematically, it's 1-1/e. It comes from the solution of the 1st order ordinary differential equation, which can be found by the inverse Laplace transform of the identified transfer function (identified model).

Practically, this time constant (Tau) can conviniently determine when the system reaches the +/-5% band (3 Tau) and +/-2% band (4 Tau) of the steady state.

Edit: corrected question #.

A1) You are correct, you can't tell. Real life systems are usually nonlinear, time-variant, and of higher order, so there's no way to know for sure the order of the system. But it doesn't matter since we're just choosing a model to fit our data. Looking at your data, I think you could fit it to a FOPDT (first order plus dead time) model. A second order system does not necessarily have to have oscillations. If the poles are both on the real axis, you will observe no oscillations in the step response.

A2) As someone already pointed out, 63% is not arbitrary but in fact comes from the very definition of the time constant: the time constant is the time it takes for the system output to reach 63% of it's final steady state value.

A way to differentiate an over damped second order from a first order is the slope at t=0. The latter has a positive slope while a second order is flat

Depending on how in depth you want to go there is a book called "Robust Adaptive Control" that is essentially a Bible on adaptive/system ID techniques.

There are a few methods that are viable. I like recursive least squares approaches because they are mathematically intuitive, flexible, and dont require something called the "persistent excitation" condition to be satisfied to ensure convergence.

https://en.wikipedia.org/wiki/Recursive_least_squares_filter

It probably doesn't matter in this case, but one important difference between first order systems and real-pole second order systems is that the step response of second order systems have smooth time derivatives, unlike first order systems which produce "sharp corners" when a step is applied. That difference is easily proven with the initial value theorem.

Another thing that may be important to note is that the 63% rule should be applied based on when the response approaches 63% of the total variation, not final value. That is, subtract the initial value (before input applied) from the final value and compute based on 63% of that.

Why is there a setpoint in your data plot? Shouldn't it be an open loop measurement, so no reference setpoint. Also are the stars the only actual data, because they do not seem to be evenly spaced in time, otherwise you could have calculated a frequency response function using a fast Fourier transform. Unevenly spaced data might also make fitting a model more complicated.

For Q1 indeed when you see an oscillation in the impulse response the system has to be at least of order two or higher (requires complex conjugate poles). However 2nd or higher order systems do not have to oscillate if it is over critically damped. But if this is indeed an impulse response then your system should at least be 2nd order, namely an integrator (otherwise you can't get a nonzero steady state value) and normal pole.

You would not necessarily see oscillations even if the system is second order if it is critically damped or overdamped. I dont know how you would do system id on that though, you could try modeling it and using least squares/ non linear least squares to estimate system parameters, but that would take a lot of work to figure out how to do if you dont get that already. MY knowledge is limited there.

Differential equations have solutions that are often times involving exponential functions, 'e^(-alpha*time). when the exponent reaches one, the value is diminished by 63%. So thats where it comes from. It is arbitrary, but it gives you clues about your system by telling you the decay rate.