I have lots of Li-ion batteries on hand. They are nominal 3.7 volts, but 4.2 volts when fully charged.
The ESP8266 will run on from 1.7 to 3.6 volts, but 3.3 volts is the idea. Given the max rating of 3.6 volts, just hooking it to a battery will not do. The rub though is the current demand.
You want a low dropout regulator (otherwise you would just usse something like an LM317).
The pinout is GND, Vin, Vout.
I bought some MCP1700-3302E LDO regulators in a handy TO92 package with the idea of using them with the ESP8266. They won't cut it. They have a maximum current output of 250 mA, which will be enough much of the time, but not always and you will see flakey operation, failure to boot, and such.
You can wire two in parallel, but this is bush league. There is no guarantee that they will be matched well enough to load balance, one will take most of the load, going to its limit and the other will fill in. This will tend to overload one of the pair and probably lead to premature failure.