Yes, in the simplest sense capacitors store charge. However there are no simple rules for using any component. An understanding of the math behind the devices is very beneficial.
Hydraulic analog: A thin tube
Mechanical analog: Friction
Relationship between voltage and current: V = IR (Ohm's law)
A resistor produces a voltage directly proportional to current flow.
Hydraulic analog: A tube with a diaphragm stretched across the middle.
Mechanical analog: The ideal spring.
Relationship between voltage and current:
For capacitors, voltage is proportional to the integral of current, or charge: v(t) = 1/C ∫ i(t) dt = Q/C. The saying is that voltage "lags" current by 90 degrees.
Capacitors operate because of the electrostatic force between adjacent insulated conductors. When you force electrons into one side, they repel electrons on the other side (and vice-versa).
They act like a conductor to high-frequency AC and an insulator to DC.
Hydraulic analog: A long pipe carrying a large mass of water
Mechanical analog: Any object with considerable momentum, such as a weight or flywheel.
Relationship between voltage and current:
For inductors, voltage is proportional to the derivative of current: v(t) = L di(t)/dt. The relationship between voltage and current is opposite of that in a capacitor; voltage "leads" current by 90 degrees.
Inductors are generally coils of wire which store energy in a magnetic field. They "oppose" changes in current. They block high-frequency AC current while conducting DC.
It is important to note that abrupt changes in current result in huge voltage spikes. This can be used advantageously (as in boost/buck converters) or it might destroy other devices if not accounted for. Electric motors are inductive loads. When a coil in the motor breaks electrical contact (which occurs many times each rotation), it generates a voltage spike known as back-emf. In order to protect the device driving the motor, diodes are often used to short this current either across the motor's terminals or to VCC/ground.
Ohm's law can be generalized to apply to capacitors and inductors: V = IZ, where V, I, and Z are all complex values. AC voltage might be represented as v(t) = Vexp(ωt+φ) (recall euler's formula), where ω=2πf is angular frequency and φ is phase.
Simple combinations of these passive components are identified by the letters representing their values: R - resistor, L - Inductor, C - capacitor.
The circuits you are describing in your post regarding capacitors are generally "RC" circuits.
These are low-pass or high-pass filters. When you see a big electrolytic capacitor from voltage supply to ground it is functioning as a low-pass filter, shorting high-frequency noise to ground and smoothing out voltage spikes/dips.
High-pass filters have the resistor and capacitor flipped. The capacitor will allow AC current to flow, but will block DC entirely.
The -3dB cutoff frequency for both RC high-pass and low-pass circuits is 1/(2πRC). The signal is attenuated by 20dB/decade for frequencies beyond this point.
When choosing coupling/decoupling or filter capacitors, you need to consider the resistance of the source/load and find a value of C that produces sufficient attenuation at the required frequency.
RL circuits can be used as high-pass or low-pass filters much like RC circuits, but arranged differently. While the capacitor blocks DC and pass AC, inductors have the opposite effect; blocking high-frequency AC and passing DC.
LC circuits resonate. Going back to the mechanical model, it's like a large mass (the inductor) on the end of a spring (the capacitor). If you pull the weight and release it, it will continue to bounce for a while at a frequency determined by the mass of the object and the strength of the spring. They have a spike in response at one particular frequency, accompanied by an abrupt shift in phase.
RLC circuits are "damped" resonators. It's like the mass-spring system with friction.
When analyzing circuits, keep in mind Kirchhoff's voltage/current laws and break the circuit into smaller chunks by considering the relative magnitudes of the part's values.