Showing posts with label orders. Show all posts
Showing posts with label orders. Show all posts

Sunday, March 25, 2012

Deleting lots of records

Apparently, deleting 7,000,000 records from a table of about 20,000,000 is not advisable. We were able to take orders at 8:00AM, but not at 7:59.

So, what's the best way of going about deleting a large number of records? Pretty basic lookup table, no relationships with other tables, 12 or so address-type fields, 4 or 5 simple indexes. I can take it down for a weekend or night, if needed.

DTS the ones to keep to another table, drop the old and rename the new table?
Bulk copy out, truncate and bring back in?
DTS to text, truncate and import back?
Other ways?

Never worked with such a large table and need a little experienced guidance.

Thanks for the helpIf you can stop any access to the table then take the indexes off, there's a special command that "keeps" the index but removes from the table, I forgot what its called. Then do the delete. You can prob turn off the transaction logging as well...although I'm not sure about that.

Sunday, March 11, 2012

Deletes through Replication

I have an order header we'll call HEADER and detail we'll call DETAIL
(catchy names !!)
When I replicate new orders, I use filters:
The HEADER article filter is like:
Select * from HEADER
where ordr_dte >= '1998-01-01'
then
The DETAIL filter is like:
Select * from DETAIL
where exists (SELECT * FROM HEADER WHERE DETAIL.ord_num = HEADER.ord_num
and (HEADER.ordr_dte >= '1998-01-01' ))
This makes sure I only get DETAILS that match filtered HEADERS.
Everything's fine so far. Subscriber matches Publisher
Now, our order processing application sometimes deletes order records. Not
very often, but if a transaction is backed out before being posted or
something.
Anyway, the problem is that on the Publisher, the application deletes the
HEADER record, which gets deleted on the Subscriber .... FINE
Then the DETAIL gets deleted on the Publisher .... BUT ... since my
filter only deals with DETAILS that have an associated HEADER, and the HEADER
doesn't exist any more, I'm left with an orphan DETAIL record on the
subscriber.
What should I do ? Is there a better way to set this up to start with ?
if you are using merge replication have a look at join filters.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
"HomeBrew" <HomeBrew@.discussions.microsoft.com> wrote in message
news:29A8C1EA-42EC-40DE-873F-5F01C2FEBAFE@.microsoft.com...
> I have an order header we'll call HEADER and detail we'll call DETAIL
> (catchy names !!)
> When I replicate new orders, I use filters:
> The HEADER article filter is like:
> Select * from HEADER
> where ordr_dte >= '1998-01-01'
> then
> The DETAIL filter is like:
> Select * from DETAIL
> where exists (SELECT * FROM HEADER WHERE DETAIL.ord_num =
HEADER.ord_num
> and (HEADER.ordr_dte >= '1998-01-01' ))
> This makes sure I only get DETAILS that match filtered HEADERS.
> Everything's fine so far. Subscriber matches Publisher
> Now, our order processing application sometimes deletes order records. Not
> very often, but if a transaction is backed out before being posted or
> something.
> Anyway, the problem is that on the Publisher, the application deletes the
> HEADER record, which gets deleted on the Subscriber .... FINE
> Then the DETAIL gets deleted on the Publisher .... BUT ... since my
> filter only deals with DETAILS that have an associated HEADER, and the
HEADER
> doesn't exist any more, I'm left with an orphan DETAIL record on the
> subscriber.
> What should I do ? Is there a better way to set this up to start with ?
|||I'm just using regular Transactional Replication.
Is there some better filter logic I can use to avoid orphans ?
"Hilary Cotter" wrote:

> if you are using merge replication have a look at join filters.
> --
> Hilary Cotter
> Looking for a SQL Server replication book?
> http://www.nwsu.com/0974973602.html
> "HomeBrew" <HomeBrew@.discussions.microsoft.com> wrote in message
> news:29A8C1EA-42EC-40DE-873F-5F01C2FEBAFE@.microsoft.com...
> HEADER.ord_num
> HEADER
>
>
|||Yes, but could you post your schema for both tables?
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
"HomeBrew" <HomeBrew@.discussions.microsoft.com> wrote in message
news:582163AF-EEC9-4633-B723-70D2B72C9E1A@.microsoft.com...[vbcol=seagreen]
> I'm just using regular Transactional Replication.
> Is there some better filter logic I can use to avoid orphans ?
> "Hilary Cotter" wrote:
Not[vbcol=seagreen]
the[vbcol=seagreen]
my[vbcol=seagreen]
with ?[vbcol=seagreen]
|||Here are the schemas for the "HEADER" and "DETAIL" tables. (1 Header to
Many Details)
Below them are the snippets from my Replication Row Filters. These tables
are part of a package, so I can't make the Detail get deleted before the
Header.
Schema for PROORD_M ("HEADER" Table, I removed many irrelevant fields in
the interest of space)
CREATE TABLE [dbo].[PROORD_M] (
[ORD_NUM] [char] (8) NOT NULL ,
[CTG_DTE] [datetime] NOT NULL ,
[ORD_CTG] [char] (6) NOT NULL ,
[NUM_OPS] [char] (8) NOT NULL ,
[CTM_NBR] [char] (12) NOT NULL ,
[CTG_DTE2] [decimal](8, 0) NOT NULL ,
[ORD_CTG2] [char] (6) NOT NULL ,
[ORD_STA] [char] (1) NOT NULL ,
[ORD_NUM2] [char] (8) NOT NULL ,
[PO_NUM] [char] (20) NULL ,
[ORDR_DTE] [datetime] NULL ,
[SHP_DTE] [datetime] NULL ,
[ORD_TYPE] [char] (1) NULL
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[PROORD_M] ADD
CONSTRAINT [PROORD_M_K1] PRIMARY KEY CLUSTERED
([ORD_NUM]) WITH FILLFACTOR = 90 ON [PRIMARY] ,
CONSTRAINT [PROORD_M_K2] UNIQUE NONCLUSTERED
([CTG_DTE],
[ORD_CTG],
[NUM_OPS]) WITH FILLFACTOR = 90 ON [PRIMARY] ,
CONSTRAINT [PROORD_M_K4] UNIQUE NONCLUSTERED
([ORD_STA],
[ORD_NUM2]) WITH FILLFACTOR = 90 ON [PRIMARY]
++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++
Schema for PROOLN_M ("DETAIL" Table, I removed many irrelevant fields in
the interest of space)
CREATE TABLE [dbo].[PROOLN_M] (
[ORD_NUM] [char] (8) NOT NULL ,
[ORD_SPSQ] [char] (6) NOT NULL ,
[ORD_SEQ] [char] (5) NOT NULL ,
[SHP_CTM] [char] (12) NOT NULL ,
[ITM_NUM2] [char] (10) NOT NULL ,
[ITM_NUM3] [char] (10) NOT NULL ,
[ORD_NUM3] [char] (8) NOT NULL ,
[OLN_STA] [char] (2) NULL ,
[ITM_NUM] [char] (10) NULL
[QTY_ORD] [int] NULL ,
[QTY_SHP] [int] NULL
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[PROOLN_M] ADD
CONSTRAINT [PROOLN_M_K1] PRIMARY KEY CLUSTERED
([ORD_NUM],
[ORD_SPSQ],
[ORD_SEQ]) WITH FILLFACTOR = 90 ON [PRIMARY]
++++++++++++++++++++++++++++
Row filter for PROORD ("HEADER" Table)
SELECT <published_columns> FROM <<PROORD_M>>
WHERE ordr_dte >= '1998-01-01' or ordr_dte is null
++++++++++++++++++++++++++++
Row filter for PROOLN ("DETAIL" Table)
SELECT <published_columns> FROM <<PROOLN_M>>
WHERE exists
(SELECT * FROM AdvDbPrd.dbo.proord_m WHERE
prooln_m.ord_num = proord_m.ord_num
and (ordr_dte >= '1998-01-01' or ordr_dte is null))

Deleteing rows

I have to delete 30K rows that were inserted into the orders table by
mistake.
The orderid field is used as a FK on many other tables so the delete takes
forever.
Is there any way to increase the performance of this delete?
In this case I know there will not be any associated FKs that reference
these orderids in other tables because the insert I am seeking to undo was
made only to the orders table. Therefore no orphans will be produced (i.e.,
no RI violation) upon delete.
Is it possible to perform the delete without constraint checking? If not,
is there anything I can do to speed up the process?
ThanksDo you have indexes on the FK columns in the referencing tables? That can
speed up such an operation significantly.
You can disable the FK constraint (see ALTER TABLE), but that disabling
applies for all connections, so make sure you are alone on the database
while doing so, if you want to take that route.
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
"David F" <davef@.nksj.ru> wrote in message
news:uS8KQd5AEHA.3348@.TK2MSFTNGP11.phx.gbl...
> I have to delete 30K rows that were inserted into the orders table by
> mistake.
> The orderid field is used as a FK on many other tables so the delete
takes
> forever.
> Is there any way to increase the performance of this delete?
> In this case I know there will not be any associated FKs that reference
> these orderids in other tables because the insert I am seeking to undo was
> made only to the orders table. Therefore no orphans will be produced
(i.e.,
> no RI violation) upon delete.
> Is it possible to perform the delete without constraint checking? If not,
> is there anything I can do to speed up the process?
> Thanks
>|||Thanks Tibor.
So it looks like I will not have to DROP the FK, just disable and then
reenable like:
--disable FK
ALTER TABLE child NOCHECK CONSTRAINT fk_id
--re-enable FK
ALTER TABLE child CHECK CONSTRAINT fk_id
Thanks
"Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote in
message news:OTRoxf5AEHA.1028@.TK2MSFTNGP11.phx.gbl...
> Do you have indexes on the FK columns in the referencing tables? That can
> speed up such an operation significantly.
> You can disable the FK constraint (see ALTER TABLE), but that disabling
> applies for all connections, so make sure you are alone on the database
> while doing so, if you want to take that route.
> --
> Tibor Karaszi, SQL Server MVP
> http://www.karaszi.com/sqlserver/default.asp
>
> "David F" <davef@.nksj.ru> wrote in message
> news:uS8KQd5AEHA.3348@.TK2MSFTNGP11.phx.gbl...
> takes
was
> (i.e.,
not,
>|||Yes, but did you check the indexes first? Having indexes on a FK column is
often crucial for reasonable performance when doing update and delete in the
referenced table. And not only that, these indexes can help your join
operations significantly (you often join over primary key - foreign key
relationships).
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
"Dave" <dave@.nodomain.tv> wrote in message
news:eZjbGj6AEHA.3256@.TK2MSFTNGP09.phx.gbl...
> Thanks Tibor.
> So it looks like I will not have to DROP the FK, just disable and then
> reenable like:
> --disable FK
> ALTER TABLE child NOCHECK CONSTRAINT fk_id
> --re-enable FK
> ALTER TABLE child CHECK CONSTRAINT fk_id
> Thanks
>
> "Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote
in
> message news:OTRoxf5AEHA.1028@.TK2MSFTNGP11.phx.gbl...
can
reference
> was
> not,
>

Deleteing rows

I have to delete 30K rows that were inserted into the orders table by
mistake.
The orderid field is used as a FK on many other tables so the delete takes
forever.
Is there any way to increase the performance of this delete?
In this case I know there will not be any associated FKs that reference
these orderids in other tables because the insert I am seeking to undo was
made only to the orders table. Therefore no orphans will be produced (i.e.,
no RI violation) upon delete.
Is it possible to perform the delete without constraint checking? If not,
is there anything I can do to speed up the process?
ThanksDo you have indexes on the FK columns in the referencing tables? That can
speed up such an operation significantly.
You can disable the FK constraint (see ALTER TABLE), but that disabling
applies for all connections, so make sure you are alone on the database
while doing so, if you want to take that route.
--
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
"David F" <davef@.nksj.ru> wrote in message
news:uS8KQd5AEHA.3348@.TK2MSFTNGP11.phx.gbl...
> I have to delete 30K rows that were inserted into the orders table by
> mistake.
> The orderid field is used as a FK on many other tables so the delete
takes
> forever.
> Is there any way to increase the performance of this delete?
> In this case I know there will not be any associated FKs that reference
> these orderids in other tables because the insert I am seeking to undo was
> made only to the orders table. Therefore no orphans will be produced
(i.e.,
> no RI violation) upon delete.
> Is it possible to perform the delete without constraint checking? If not,
> is there anything I can do to speed up the process?
> Thanks
>|||Thanks Tibor.
So it looks like I will not have to DROP the FK, just disable and then
reenable like:
--disable FK
ALTER TABLE child NOCHECK CONSTRAINT fk_id
--re-enable FK
ALTER TABLE child CHECK CONSTRAINT fk_id
Thanks
"Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote in
message news:OTRoxf5AEHA.1028@.TK2MSFTNGP11.phx.gbl...
> Do you have indexes on the FK columns in the referencing tables? That can
> speed up such an operation significantly.
> You can disable the FK constraint (see ALTER TABLE), but that disabling
> applies for all connections, so make sure you are alone on the database
> while doing so, if you want to take that route.
> --
> Tibor Karaszi, SQL Server MVP
> http://www.karaszi.com/sqlserver/default.asp
>
> "David F" <davef@.nksj.ru> wrote in message
> news:uS8KQd5AEHA.3348@.TK2MSFTNGP11.phx.gbl...
> > I have to delete 30K rows that were inserted into the orders table by
> > mistake.
> >
> > The orderid field is used as a FK on many other tables so the delete
> takes
> > forever.
> >
> > Is there any way to increase the performance of this delete?
> >
> > In this case I know there will not be any associated FKs that reference
> > these orderids in other tables because the insert I am seeking to undo
was
> > made only to the orders table. Therefore no orphans will be produced
> (i.e.,
> > no RI violation) upon delete.
> >
> > Is it possible to perform the delete without constraint checking? If
not,
> > is there anything I can do to speed up the process?
> >
> > Thanks
> >
> >
>|||Yes, but did you check the indexes first? Having indexes on a FK column is
often crucial for reasonable performance when doing update and delete in the
referenced table. And not only that, these indexes can help your join
operations significantly (you often join over primary key - foreign key
relationships).
--
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
"Dave" <dave@.nodomain.tv> wrote in message
news:eZjbGj6AEHA.3256@.TK2MSFTNGP09.phx.gbl...
> Thanks Tibor.
> So it looks like I will not have to DROP the FK, just disable and then
> reenable like:
> --disable FK
> ALTER TABLE child NOCHECK CONSTRAINT fk_id
> --re-enable FK
> ALTER TABLE child CHECK CONSTRAINT fk_id
> Thanks
>
> "Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote
in
> message news:OTRoxf5AEHA.1028@.TK2MSFTNGP11.phx.gbl...
> > Do you have indexes on the FK columns in the referencing tables? That
can
> > speed up such an operation significantly.
> >
> > You can disable the FK constraint (see ALTER TABLE), but that disabling
> > applies for all connections, so make sure you are alone on the database
> > while doing so, if you want to take that route.
> >
> > --
> > Tibor Karaszi, SQL Server MVP
> > http://www.karaszi.com/sqlserver/default.asp
> >
> >
> > "David F" <davef@.nksj.ru> wrote in message
> > news:uS8KQd5AEHA.3348@.TK2MSFTNGP11.phx.gbl...
> > > I have to delete 30K rows that were inserted into the orders table by
> > > mistake.
> > >
> > > The orderid field is used as a FK on many other tables so the delete
> > takes
> > > forever.
> > >
> > > Is there any way to increase the performance of this delete?
> > >
> > > In this case I know there will not be any associated FKs that
reference
> > > these orderids in other tables because the insert I am seeking to undo
> was
> > > made only to the orders table. Therefore no orphans will be produced
> > (i.e.,
> > > no RI violation) upon delete.
> > >
> > > Is it possible to perform the delete without constraint checking? If
> not,
> > > is there anything I can do to speed up the process?
> > >
> > > Thanks
> > >
> > >
> >
> >
>