SQLite Forum

SQLITE count(*) help appreciated
Login

SQLITE count(*) help appreciated

(1) By Gebruikersnaam on 2024-07-24 09:31:04 [link] [source]

To get data from sqlite from last year I'm using this one:

select count(*), * from DayData where TimeStamp > 1672532454 and TimeStamp < 1704064854 and Power >= 500;

Can you help me change the select criteria to find out the number of days where Power >= 500 happens more than 25 times per day.

(2) By drh on 2024-07-24 10:14:10 in reply to 1 [link] [source]

NB: The following is untested. I'm just writing SQL off the top of my head. The examples below might not work and might contain syntax errors.

To see the number of times that Power>=500 for each day:

SELECT date(Timestamp,'unixepoch') AS date, count(*) AS cnt
  FROM DayData
 WHERE Power>=500
 GROUP BY date
 ORDER BY date;

Using the above as a subquery to compute the number of days on which cnt exceeds 25:

SELECT count(*) FROM (
    SELECT date(Timestamp,'unixepoch') AS date, count(*) AS cnt
      FROM DayData
     WHERE Power>=500
     GROUP BY date
) WHERE cnt>50;

(5.1) By Gebruikersnaam on 2024-07-24 20:57:18 edited from 5.0 in reply to 2 [link] [source]

I started with this :

select count(*), * from DayData where TimeStamp > 1672531200 + 86400 * 1 and TimeStamp < 1672617600 + 86400 * 0 and Power >= 500;

select count( * ), * from DayData where TimeStamp > 1672531200 + 86400*1 and TimeStamp < 1672617600 + 86400 * 2 and Power >= 500;

to select for 1st and second January 2023

count(*) TimeStamp Serial TotalYield Power PVoutput
18 1672656000 2130121705 49194377 528 1

On 1 July 2023

count(*) TimeStamp Serial TotalYield Power PVoutput
163 1688273700 2130121705 52375823 516 1

Your example shows the data from today I suppose.

How can it be changed to itinerate through the database and display a column of dates and count(*) outcomes? (or export it as CSV)

You might ask why I want these figures. Well I intend to mount a hotfill close-in boiler (400W)and fill it with cold water and heat it for immediate warm water from the tap. I want to know the chance for each day of the year that PV power might be able to heat the water inside the boiler to a reasonable temperature. PV output value is measured every 5 minutes during production time (daylight) To heat the water I need 1 hour of input plus heating the used water from the tap, assuming about 25 values of > 500 Watt.

(3) By midijohnny on 2024-07-24 10:58:19 in reply to 1 [link] [source]

I think you should be able to use GROUP and HAVING here:

SELECT timestamp, COUNT(*) AS count
FROM daydata
WHERE power>500
GROUP BY timestamp
HAVING count > 25;

You can provide your timestamp criteria as an 'AND' clause in the 'WHERE' section.

Test case below (it generates a bunch of random rows for testing).

-- DROP TABLE IF EXISTS daydata;
CREATE TABLE daydata(
	daydate_id INTEGER PRIMARY KEY NOT NULL, 
	timestamp TEXT NOT NULL, -- Note: using date strings, rather that epoch - adjust as necessary.
	power INTEGER NOT NULL
);

INSERT INTO daydata(timestamp, power)
WITH RECURSIVE 
	_vars(start_date, end_date, min_power, max_power, max_i) AS (
		SELECT
			DATE('now', '-1 year', 'start of year'),
			DATE('now', '-1 year', 'start of year', '+1 year', '-1 days'),
			0,1000,
			10000),
	_rnd_date(d) AS (
		SELECT DATE(start_date, printf('+%d days', 
					abs(random() %( 1 + (julianday(end_date) - julianday(start_date))))))
		FROM _vars),
	_rnd_power(p) AS (
		SELECT min_power + abs(random() % (1+(max_power-min_power)))
		FROM _vars),
	_loop(i) AS (
			SELECT max_i FROM _vars 
				UNION ALL
			SELECT i-1 FROM _loop 
			LIMIT (SELECT MAX(1,IFNULL(max_i,1)) FROM _vars)	
	)
SELECT d,p FROM _loop
JOIN _rnd_date
JOIN _rnd_power;

(4) By midijohnny on 2024-07-24 11:07:48 in reply to 3 [link] [source]

If you just want the count, rather than the summary, wrap it up as follows:

WITH
	_summary(timestamp, count) AS (
		SELECT timestamp, COUNT(*) AS count
		FROM daydata
		WHERE power>500
		GROUP BY timestamp
		HAVING count > 25
	)
SELECT COUNT(*) AS no_of_days from _summary;

Note: the test-case I provided doesn't guarantee that you will get data that will satisfy the criteria (but I tested with lower counts, seems to work).

(6.1) By Gebruikersnaam on 2024-07-24 20:34:32 edited from 6.0 in reply to 4 [link] [source]

In this case I need to have the sum of a timestamp + the next ones up till timestamp + 86400. After that skip to the next timestamp + every timestamp + 86400 et cetera.

I only have timestamp, however I want to count the total score of every day of the year.

(7.2) By midijohnny on 2024-07-24 14:36:10 edited from 7.1 in reply to 6.0 [link] [source]

To clarify - do you need to 'pad' days where are no readings at all?

That is: do you need a comprehensive list of days within a range?
Then consider a LEFT JOIN against a CTE that generates day entries for the range given.

Something like:

WITH RECURSIVE
	_vars(start_date, end_date) AS (
		SELECT
			DATE('now', '-1 year', 'start of year'),
			DATE('now', '-1 year', 'start of year', '+1 year', '-1 days')
		),
	_generator(start_date, end_date, iter_date) AS (
		SELECT start_date, end_date, start_date FROM _vars	
			UNION ALL
		SELECT start_date, end_date, DATE(iter_date, '+1 days')
		FROM _generator
		WHERE DATE(iter_date) < DATE(end_date)
		LIMIT 1000 --- safety valve
	),
	_summary(timestamp, count) AS (
		SELECT timestamp, COUNT(*) AS count
		FROM daydata
		WHERE power>500
		GROUP BY timestamp
		HAVING count > 25
	)
SELECT iter_date AS timestamp, IFNULL(count,0) AS "READNG COUNT WHERE POWER>500 FOR THIS DATE" FROM _generator
LEFT JOIN _summary ON timestamp=iter_date;

Example result:

timestamp READNG COUNT WHERE POWER>500 FOR THIS DATE
[...] [...]
2023-07-09 0
2023-07-10 0
2023-07-11 0
2023-07-12 0
2023-07-13 0
2023-07-14 29
2023-07-15 0
2023-07-16 0
2023-07-17 0
2023-07-18 0

[...]

(8.2) By Gebruikersnaam on 2024-07-25 17:53:00 edited from 8.1 in reply to 7.2 [source]

Deleted

(9.14) By Gebruikersnaam on 2024-07-25 22:24:17 edited from 9.13 in reply to 8.1 [link] [source]

I have come this far:

sqlite3 -separator ' ' SBFspot.db "select Power,DATE(TimeStamp, 'unixepoch') from DayData where Power >= 500 and DATE(TimeStamp, 'unixepoch')< '2024-01-01' and DATE(TimeStamp, 'unixepoch')>= '2023-01-01';" > results.csv

 2023-01-02 504
 2023-01-02 516
 2023-01-02 552
 2023-01-02 540
 2023-01-02 552
 2023-01-02 540
 2023-01-03 612
 2023-01-03 516
 2023-01-04 528
 2023-01-04 564
 2023-01-06 552

awk '{print $1}' results.csv | sort | uniq -c > results_counted.csv

 14 2023-01-01
 18 2023-01-02
  2 2023-01-03
  2 2023-01-04
  1 2023-01-06

awk '{if($1>50) print $1,$2;}' results_counted.csv > dates-with-sufficient-energy.csv

 56 2023-01-19
 51 2023-01-28
 68 2023-02-04
 65 2023-02-06
 55 2023-02-07
 56 2023-02-08
 74 2023-02-09
 79 2023-02-10
 54 2023-02-11
 70 2023-02-13
 67 2023-02-14
 75 2023-02-15
 64 2023-02-19
 71 2023-02-22

(10.1) By Gebruikersnaam on 2024-07-27 20:08:51 edited from 10.0 in reply to 9.14 [link] [source]

The solution has been handed to me on the domoticz forum. https://www.domoticz.com/forum/viewtopic.php?p=318864#p318864

select count() from (select DATE(TimeStamp, 'unixepoch') day, count() c from DayData where Power >= 500 and DATE(TimeStamp, 'unixepoch')< '2024-01-01' and DATE(TimeStamp, 'unixepoch')>= '2023-01-01' group by day) where c>50;

It renders the # of days where there is sufficient energy to heat the close-in boiler with sun power.